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Software product lines (SPL) are a method for the development of variant-rich software systems. 
Compared to non-variable systems, testing SPLs is extensive due to an increasingly amount of possi¬ 
ble products. Different approaches exist for testing SPLs, but there is less research for assessing the 
quality of these tests by means of error detection capability. Such test assessment is based on error 
injection into correct version of the system under test. However to our knowledge, potential errors in 
SPL engineering have never been systematically identihed before. 

This article presents an overview over existing paradigms for specifying software product lines 
and the errors that can occur during the respective specihcation processes. For assessment of test 
quality, we leverage mutation testing techniques to SPL engineering and implement the identihed 
errors as mutation operators. This allows us to run existing tests against defective products for the 
purpose of test assessment. From the results, we draw conclusions about the error-proneness of the 
surveyed SPL design paradigms and how quality of SPL tests can be improved. 

1 Introduction 

Software product line (SPL) engineering is an emerging method for the development of variant-rich 
software systems. Based on a SPL specification single products can be configured and derived. SPL 
engineering is a sysfemafic and planned process fo reuse soffware arfifacfs mosf efficienfly O. This also 
includes qualify assurance, where one of fhe mosf imporfanf ones is fesfing. Buf fesfing a SPL is differenf 
from fesfing non-variable sysfems and fhus is invesfigafed infensively |[27l [TTl IMl 1^ . Challenges in 
fesfing SPLs are fhe selection of producfs for fesfing and fhe design of fesfs from fhe SPL’s specificafion. 

Though fhere are many mefhods proposed for fesfing a producf line, unfil now, qualify assessmenf of 
fesfs was limifed fo mufafing individual producfs of fhe SPL. This approach has fwo major drawbacks: 
firsf, developers can infroduce errors on all kinds of arfifacfs, nof only on final producfs specifications. 
For beffer undersfanding we analyze differenf design paradigms for fhe specificafion of producfs from a 
SPL and fhe errors fhaf can occur during fhe respecfive design processes. From fhe resulfs, we developed 
mufafion operafors for variabilify models, and domain models fhaf mimic possible faulfs in fhese models. 

Secondly, fhe selecfion of producfs and subsequenfly ifs mufafions is biased by fhe available fesfs, 
since only producfs for which fesfs are available will be fesfed. Therefore, mufafion analysis assesses 
the quality of the tests for particular products, but not for the whole SPL. In contrast, we define a mu¬ 
tation system and operators on the domain engineering-level. This enables us to assess the test quality 
independently from the tested products. Subsequently, the test quality over the complete SPL is assessed. 

The remainder of this article is structured as follows: In Section |2] we summarize the foundations of 
SPL engineering and test assessment. In Sec. we define and classify kinds of errors. We present our 
SPL test assessment system and the evaluation of three examples in Sec. Eventually, we show related 
work in Sec.|5]and conclude in Sec.|6l 
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Figure 1: Process of SPL engineering 


Figure 2: A feature model for the eShop example. 


2 Preliminaries 

In this section, we present the foundations that our work is based on. First, we give a short introduction 
for model-based product line engineering. The second part is about mutation analysis. The third part 
deals with potential errors and mutation in software development. 

2.1 Model-based Product Line Engineering 

Individual customer expectations and the reuse of existing assets in a product’s design are two driving 
factors for the emergence of product line engineering: increasing the number of product features while 
keeping system engineering costs at a reasonable level. In terms of software engineering, a SPL is a set 
of related software products that share a common core of software assets (commonalities), but can be 
distinguished (variabilities) |[29l . 

The definition and realization of commonalities and variabilities is the process of domain engineer¬ 
ing. Actual products are built during application engineering (cf. fig. [T]l. Here, producfs are builf by 
reusing domain arfifacfs and exploiting fhe producf line variabilify. 

Like many mefhodologies, SPL engineering can be supporfed by model-based absfracfions such as 
feafure models. Feafure models offer a way fo overcome fhe aforemenfioned challenges by facilifafing 
the explicit design of global system variation points ifTSll . In consequence, variation points are not spread 
across one or multiple domain models anymore, but instead linked to one core of variability description. 

A feature model has a tree structure in which a feature can be decomposed into sub-features. Fig. 
shows an example feature model, that will also be used as an example in section]^ A parent feature can 
have the following relations to its sub-features: (a) Mandatory, child feature is required, (b) Optional: 
child feature is optional, (c) Or: at least one of the children features must be selected, and (d) Alternative: 
exactly one of the children features must be selected. Furthermore, one may specify additional (cross¬ 
tree) constraints between two features A and B: (i) A requires B: the selection of A implies the selection 
of B, and (ii) A excludes B: both features A and B must not be selected for the same product. 

A feature model captures the system’s variation points in a concise form. Its elements, however, are 
only symbols [^il. Their semantics has to be provided by mapping them to models with semantics. Such 
a mapping can be defined using an explicif mapping model. A mapping model consisfs of relations from 
feafure model elemenfs fo domain model elemenfs. We refer fo fhe fuple of feafure model, mapping 
model, and domain model as SPL specification. 

Producf models or code can be maferialized from fhe SPL specification by providing a configura¬ 
tion. A configuration assigns a valuation fo every feafure in fhe feafure model, denoting fhe presence or 
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Figure 3: Negative (a), positive (b) variability (based Figure 4: SPL Design with Negative Variability, 
on ifTSll l and delta modeling (c) 


absence of the mapped elements. The valuation must not violate the constraints imposed by the feature 
model. 

Based on this setup three paradigms have established for specifying SPLs. These paradigms will be 
briefly introduced as follows. 

2.1.1 Negative Variability 

In this case, the domain model is designed in terms of a so called 150% model. A 150% model contains 
every element that is used in at least one product configuration and, thus, subsumes every possible prod¬ 
uct Ifldll (Fig.|^). We consider the combination of a feature model, a mapping model, and a UML model 
as SPL specification. 

Each mapping in the mapping model maps a single feature to a set of transitions. Additionally, 
each mapping has a Boolean flag that indicates whether the mapped model elements are part of the 
product when the feature is selected (true) or unselected (false). Figure]^ shows an excerpt of the eShop 
specification, where parts of the feature model are depicted in the upper half and parts of the state 
machines payment process are shown in the lower half. In between, we find a mapping, denoted by a 
dotted edge, from feature ’’Credit Card” to the transition labeled as ”SelectCreditCard[]/”. 

2.1.2 Positive Variability 

In contrast to negative variability to design the domain model, positive variability starts with a minimal 
core that contains features that are common to all possible products. From this starting point additional 
features will be added by a designer (Fig.|^). 

2.1.3 Delta Modeling 

Designing products in SPL engineering using positive or negative variability is called feature-oriented. 
In contrast to these paradigms, there is another approach which is referred to as delta modeling (also 
delta-oriented programming) |[30l . Using delta modeling for the purpose of designing SPLs, two parts 
are needed. The first is a core module, that comprises a set of features that represent a valid product. 
The second part is a set of delta modules which specify changes that will be applied to the core module. 
These changes can either be the construction (add) or destruction (remove) of features (Lig.[^). 
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2.2 Mutation Analysis 

Mutation analysis (also mutation testing) as introduced by DeMillo et al. iTTOll is a error-based testing 
technique with the intended purpose to assess the quality of tests that will be applied to a system. 

The process of mutation analysis seeds errors into software by creating modified versions of the 
original software, where each created version contains one error. After that existing test cases are used to 
execute the defective versions (mutants) with the goal to distinguish the defective ones (to kill a mutant) 
from the original software. The ratio of killed mutants to generated mutants is called mutation score, that 
will be computed after the execution of all test cases. The main goal of the test designer is to achieve the 
highest possible mutation score EhlfTTl . 

Though mutation operators are applied to introduce errors, there is the chance, that the resulting 
mutant offers the same behavior like the original. This type of mutants are referred as hidden mutants. 
Although the detection of hidden mutants is an undecidable problem, hidden mutants are supposed to be 
removed from the mutation analysis before scoring is performed ifTTl . 

According to Jian and Harman ifT^ we can distinguish multiple kinds of mutants that can be created. 
The simplest ones and already mentioned are first-order mutants that have only one introduced error. 
Even if first-order mutants can be killed during the process of mutation testing, this does not guarantee 
that a combination of two (or even more) mutants will also be detected by the test suite. Such combined 
mutants are referred as higher-order mutants. 

2.3 Potential Errors and Mutations 

In mutation analysis, defective software versions are derived from a set of potential errors a human can 
make during software development. Potential errors are implemented as mutation operators, which are 
applied to the original software for introducing errors. The mutation operator’s design affects the validity 
of the resulting mutation scores and the costs for testing by means of the amount of mutants to create 
and the number of tests to execute against them. Thus, we apply the following four guiding principles 
for creating mutation operators l|5l [33l : 

1. Mutation categories should model potential error. It is important to recognize different types of 
error. In fact, each mutation operator is designed to model errors belonging to the corresponding 
error class. 

2. Only simple, first-order mutants should be generated. These mutants are produced by making ex¬ 
actly one syntactic change to the original specification. This restriction is justified by fhe coupling 
ejfect hypofhesis which says fhaf fhe fesf sefs fhaf defecf simple mufanfs will also defecf more 
complex mufanfs ll2^ . 

3. Only synfacfically and semanfically correcf mutanfs should be generated. Some mufafions may 
resull in an illegal expression, such as division by 0. Such mutanfs should nol be generated. 

4. Do nol produce foo many mutanfs. This includes some pracfical resfricfions. For example, do not 
replace a relational connector with its opposite, if for other mutants a term negation operator is 
applied, since both mutants are semantically equivalent. 

From other mutation systems Sill[121, we identified fhe following general cafegories for model- 
based mufafion operafors: 

1. Model elemenf delefion: a model designer forgels fo add a model elemenf, e.g. a fealure, a map¬ 
ping, or a fransifion 
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2. Model element insertion: a model designer inserts a superfluous model element, e.g. a feature, a 
mapping, or a transition 

3. Property ehange: a model designer ehooses a wrong value for a property of a model element, 
e.g. mandatory feature instead of optional, inverse value for a feature’s status, or wrong transition 
target. 

For eaeh model element-type, like mappings, transitions, guards, ete., one ean eheek for applieable 
eategories and implement mutation operators aeeordingly. 


3 Potential Errors in Model-Based Product Line Engineering 

In this eontribution, we foeus on errors in the feature mapping. The feature mapping has a major impaet 
on the outeome of the produet line’s materializations, however the design is eomplex and error-prone. 
We identify potential errors in a systematie way by eheeking eaeh modeling paradigm for possibilities 
to add superfluous or omit neeessary elements or ehange the value of an element’s attribute. For eaeh 
potential error we diseuss its effeets onto the materializations. 

Furthermore, aeeording to the eonsequenees of eaeh errors for affeeted produets, we assign one of 
the following four types to eaeh potential error: 

add extends the behavior of affeeted produets. 
omit restriets the behavior of affeeted produets. 
alter extends and restriets the behavior of some produets. 

mix extends, restriets, or both the behavior of affeeted produets, depending on the model’s eontents. 

Negative Variability 

In the negative variability paradigm, we identify the following model elements for potential errors from 
the feature mapping model: mappings, their attribute feature value, mapped feature, and the set of 
mapped elements. The errors whieh ean be made on these model elements and their effeets are as 
follows: 

Nl) Omitted mapping: a neeessary mapping is left out by its entirety. Mapped elements will be part of 
every produet unless they are restrieted by other features. As a result, some or all produets unrelated to 
the partieular feature will inelude superfluous behavior. Produets ineluding the mapped feature are not 
affeeted, sinee the behavior was enabled anyway. 

N2) Superfluous mapping: a superfluous mapping is added, sueh that a previously unmapped feature is 
now mapped to some domain model elements. This may also inelude adding a mapping for an already 
mapped feature, but with inverted feature value. Adding a mapping with feature value set to true results 
in the removal of elements from produets unrelated to the mapped feature. Contrary, adding a mapping 
with feature value set to false removes elements from any produet whieh the mapped feature is part of. 
In any ease the behavior of at least some produets is redueed. 

N3) Omitting a mapped element: a mapped model element is missing from the set of mapped element 
in a mapping. Subsequently, a previously mapped element will not only be available in produets whieh 
the said feature is part of, but also in produets unrelated to this feature. As a result, some produets offer 
more behavior than they should or eontain unreaehable model elements. 
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N4) Superfluously mapped element: an element is mapped although it should not be related to the feature 
it is currently mapped to. As a result the element becomes unavailable in products which do not include 
the associated feature. The product’s behavior is hence reduced. 

N5) Swapped feature: the associated features of two mappings are mutually exchanged. Subsequently, 
behavior is exchanged among the two features and thus, affected products offer different behavior than 
expected. The result is the same as exchanging all mapped elements among two mappings. 

N6) Inverted feature status: the bit-value of the feature value attribute is flipped. The mapped elements 
of the affected mapping become available to products where they should not be available. At the same 
time, the elements become unavailable in products where they should be. For example, if the feature 
value is true and is switched to false, the elements become unavailable to products with the associated 
feature and available to any product not including the said feature. Of course, other feature mappings to 
the same element(s) must still be considered. 

Positive Variability 

In SPL modeling with positive variability, a mapping is a bijection between features and modules com¬ 
posed from domain elements. Potential errors in the feature mapping models can be made at: mappings, 
mapped feature, and mapped module. We identify the following potential errors: 

PI) Omitted mapping: a necessary mapping is missing in its entirety. This appears to us to be an 
unrealistic scenario, since one can automatically check for all modules being mapped to some feature. 
But if we consider the case of a missing mapping, products with the associated feature would be missing 
the modules functionality. 

P2) Superfluous mapping: a superfluous mapping is added. Similar to the above, this is an unrealistic 
scenario for same reason: all modules should be mapped exactly once. In a model-based environment, 
this check should be easily automatable. However, if adding a superfluous mapping is possible, more 
behavior becomes enabled in products containing the mapping’s feature. 

P3) Swapped modules: the associated modules of two mappings are mutually exchanged. As a result, 
all products containing one of the two features, but not the other, offer not the expected behavior. In 
contrast, all products containing none or both features behave as expected. 

P4) Swapped features: the associated features of two mappings are mutually exchanged. The result is 
the same as above for swapped modules. 

Delta Modeling 

For other paradigms, like delta-modeling ||30l, we make similar observations. In contrast to positive 
variability models, delta-oriented variability models start from an actual core product, instead of a base 
module. From this on, only the differences from one product to another are defined by deltas. In delta¬ 
modeling, mapping multiple features to the same delta is allowed. A delta may add elements to and 
remove elements from the core product at the same time. As potential points of errors in delta-modeling 
we identify deltas, a delta’s set of mapped features, its set of removed elements from the base product, 
and its set of added elements. 

Dl) Omitted delta: the product line model misses an entire delta definition. Products containing features 
of the missing delta may lack behavior or offer to much of it. This depends on whether the delta removes 
and/or adds elements to the base product. 
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D2) Superfluous delta: an unnecessary delta is added. As a result, products containing the associated 
feature(s) will offer additional behavior. Also, affected products might lack behavior if the delta removes 
elements. 

D3) Omitted feature: a necessary feature from the set of mapped features is missing. If no feature is 
left, the delta is not mapped at all which can be statically verified. If otherwise the set still contains at 
least one feature, any product containing the current set of mapped features but not the missing feature, 
offers too much or too few behavior. In some cases, the set of mapped features and the affected elements 
may collide with another delta, which is again statically verifiable. 

D4) Superfluous feature: an addifional feafure is added fo a delfa’s already complefe sef of mapped 
feafures. As a resulf, fhe della will be available in less producls. If Ihe added feafure mulually excludes 
one of fhe already mapped feafures, fhe delfa will be applicable lo no producf al all. A sfalic check can be 
used fo validale lhal a sef of feafures is salisfiable by some producf. Only producls conlaining fhe correcl 
sef of feafures bul nol fhe superfluous, are affecled by Ihis error. Affecled producls may offer more or 
less behavior. 

D5) Omitted base element: a della’s sef of base elemenls is missing an elemenl. In consequence, loo 
few elemenls are removed from fhe core producf by Ihis delfa fo malch fhe producl’s specificalion. Thus 
any producf conlaining fhe feafures from Ibis della offers loo much behavior. 

D6) Superfluous base element: a della’s sef of base elemenls conlains addifional elemenls. This will 
remove more elemenls lhan necessary from fhe producls affecled by Ibis della. Hence, Ihese producls 
offer loo few behavior. 

D7) Omitted delta element: an elemenl from fhe sel of della elemenls in a della is missing. As a resull, 
any producf conlaining fhe della’s feafures offers less behavior lhan specified. 

D8) Superfluous delta element: an elemenl from a della’s sel of della elemenls is missing. In conse¬ 
quence, Ihe producls conlaining Ihe della’s feafures offer more behavior lhan specified. 

4 Product Line Test Assessment 

As laid oul in section olher errors can be made in model-based SPL engineering lhan in conlrasl lo 
non-variable systems engineering. Furlhermore, currenl lesl design melhods and coverage criteria are 
nol prepared lo deal wilh Ihese errors. To show Ihe validily of our argumenl, we propose a mulalion 
system for SPLs. Il is specifically designed lo assess lesl qualify, by means of error detection capabilily 
(EDC), for Ihe whole producf line ralher lhan for single systems. Bul mulalion systems for SPLs need 
novel mulalion operators. The reason for Ibis is Ihe separation of concerns in SPL engineering, where 
variabilily and domain engineering are splil into differenl phases and models. 

Mulalion operators defined for non-varianl systems cannol infer mulanls including modules from 
olher producls, since Ihis information is only available during domain engineering. However, we expecl 
a high-qualily lesl suite to delecl such errors. Hence, we also propose new mulalion operators based on 
Ihe potential errors, we identified in section For conciseness, we only consider potential errors from 
negative variabilily modeling for implemenlalion. 

4.1 Mutation System for SPLs 

Performing mulalion analysis on SPL lesls is differenl from non-varianl system lesls, since in conlrasl 
to conventional mulalion systems, a mulaled SPL specification is nol execulable per se. Thus, testing 
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Figure 5: Mutation Process for SPLs 


cannot be performed until a decision is made towards a set of products for testing. This decision depends 
on the SPL test suite itself, since each test is applicable to just a subset of products. 

In Figure]^ we depict a mutation process for assessing SPL test suites, which addresses this issue. 
Independently from each other, we gain (a) a set of SPL specification mutants by applying mutation 
operators to the SPL specification and identify (b) a set of configurations describing the applicable prod¬ 
ucts for testing. We apply every configuration in (b) to every mutant in (a), which returns a new set of 
product specification mutants. Any mutant structurally equivalent to the original product specification is 
immediately removed and does not participate in the scoring. The specification mutants are easily mate¬ 
rialized into product mutants and finally, tests are executed. Our mutation scores are based on the SPL 
specifcation mutants, hence we established bidirectional traceability from any SPL specification mutant 
to all its associated product mutants and back again. If a product mutant is killed by a test, we backtrack 
its original SPL specification mutant and flag it as killed. The hnal mutation score is then calculated 
from the killed and the overall number of SPL specihcation mutants. 

4.2 SPL Mutation Operators 

Here, we present mutation operators for feature mapping models with negative variability. Furthermore, 
we enrich the mutation system by standard state machine operators and apply them on domain-level as 
well. For each operator, we describe how it was identihed and its notion. Also, we discuss potentially 
invalid and hidden mutants resulting from each operator. 

4.2.1 Feature Mapping 

We design the mutation operators according to the potential errors identihed in section We do not 
consider inserting superhuous mappings as in this case it remains unclear which and how many UML 
elements should be selected for the mapping. We assume that this, if not carefully crafted, will lead to 
mostly invalid mutants. 

Delete Mapping (DMP) The deletion of a mapping will permanently enable the mapped elements, if 
they are not associated to other features that constrain their enabledness otherwise. In our examples, no 
invalid mutants were created. However, for product lines that make heavy use of mutual exclusion (Xor 
and excludes) this does not apply. The reason for this are competing UML elements like transitions that 
would otherwise never be part of the same product. Multiple enabled and otherwise excluding transitions 
are possibly introducing non-determinism or at least unexpected behavior. 

Some products mutants created with this operator might behave equivalent to an original product. This 
is the case for all products that include the feature for which the mapping was deleted. 
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Delete Mapped Element (DME) This operator deletes a mapped UML element from a mapping in the 
feature mapping model. It resembles the case, where a modeler forgot to map a UML element that should 
have been mapped. 

Similar to the delete mapping operator, this operator may yield non-deterministic models, where other¬ 
wise excluding transitions are concurrently enabled. Product mutants equivalent to the original product 
model can be derived, if the feature associated to the deleted UML reference is part of the product. 

Insert Mapped Element (IME) This operator inserts a new UML element to the mapping. This is the 
contrary case to the operators defined before, where mappings and UML elements were removed. How¬ 
ever, inserting additional elements is more difficult than deleting them, since a heuristic must be provided 
for creating such an additional element. We decided to copy the first UML element reference from the 
subsequent mapping. If there are no more mappings, we take the first mapping. This operator is not 
applicable if there is just one mapping in the feature mapping model. 

Again, there is a chance of creating invalid mutants: If a UML element reference is copied from a 
mutually excluded mapping, the resulting model may be invalid due to non-determinism. 

Swap Eeature (SWP) Swapping features exchanges the mapped behavior among each other. This oper¬ 
ator substitutes a mapping’s feature by the following mapping’s feature and vice versa. The last feature 
to swap is exchanged with the very first of the model. 

Non-deterministic behavior and thus invalid models may be designed by this operator. This is due to the 
fact that the mutation operator may exchange a feature from a group of mutually exclusive features by 
an unrestricted feature. In consequence, the previously restricted feature is now independent, while the 
unrestricted feature joins the mutually exclusive group. This may concurrently enable transitions which 
results in non-deterministic behavior. 

Change Eeature Value (CEV) This operator flips fhe feafure value of a mapping. A modeler may have 
selected the wrong value for this boolean property of each mapping. 

The operator must not be applied to a mapping, if there is a second mapping with the same feature, but 
different feature value. Otherwise, there will be two mappings for the same feature with the same feature 
value, which is not allowed for our feature mapping models. 

This operator may yield invalid mutants, if it is applied to a mapping that excludes another feature. In 
that case, two otherwise excluding UML elements can be present at the same time, which may result in 
invalid models, e.g. two default values assigned to a single variable or concurrently enabled transitions. 

4.2.2 UML State Machine 

In the past 20 years, many mutation operators for transition-based systems were defined lfT^l2^ l2ll^. 
Here, we limit ourselves to the design of operators based on transitions as these may have the strongest 
impact on the behavior of the SUT. We do not design operators that can be mimicked by the combination 
of two of them. In particular, we do not consider the exchange of an element by another, since this can 
easily be mimicked by removing and inserting the removed element at another point in the model. 

We identified five fargefs for mutation: (i) remove the entire transition, change its (ii) target state, as 
well as mutating its (iii) triggers, (iv) guard, and (v) effect. The latter three can be mutated according 
to the three defined cafegories delete, add and change. Though in this contribution omitted the category 
change for simplicity. 

For all mutants created by the here presented operators, there is a chance of materializing mutants 
behaving equivalent to the original product. This is the case, when the mutated element is part of disabled 
feature. Of course, hidden mutants - if detected - will be excluded from the scoring. 
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In general, we will not apply any elass mutation to our UML state machines lfT9l . The system’s 
logic is designed in the state machine diagrams, while the classes are merely containers for variables and 
diagrams. 

Delete Transition (DTK) Deletes a transition from a region in a UML state machine. This operator might 
create invalid UML models, if not enough transitions remain on a pseudo-state (fork, join, junction, and 
choice) II 22 I p.555]. 

Change Transition Target (CTT) Changes the target of a transition to another state of the target state’s 
region. This operator is only applicable if the region has more than one state. 

Delete Effect (DEE) Deletes the entire effect from a transition. We consider sending signals to the 
environment or other components to be part of a transition’s effect, hence they are deleted as well. 

Delete Trigger (DTI) Deletes a transition’s trigger. Only a single trigger is deleted at a time, but every 
trigger is deleted once. 

Insert Trigger (ITG) Copies an additional trigger to a transition. The trigger is copied from another 
transition within the same region. This may lead to non-deterministic behavior if both transitions, the 
source transition of the trigger and the mutated transition, are outgoing transitions of the same state. 

Delete Guard (DGD) Deletes the entire guard of a transition. This may lead to non-deterministic be¬ 
havior of the state machine, if another transition is enabled simultaneously. Furthermore, in the case 
of transitions without triggers and where source and target are the same state, this operator leads to in¬ 
finite looping of the state machine over the mutated transition. The reason for this behavior is UML’s 
run-to-completion semantic, where an enabled transition without triggers is immediately traversed. 

Change Guard (CGD) Changes a guard’s term by exchanging operators or substituting boolean literals 
by their inverse. Our CGD operator supports 30 different arithmetic, relational, bitwise, compound 
assignment, and logic operators. Furthermore, literal ’’null” is exchanged by ’’this”. 

This may cause mutants with non-deterministic behavior, whenever two transition become concurrently 
enabled due to the manipulation of one of their guards. 

4.3 Evaluation 

We created three example product lines for performing a mutation analysis on them. We designed the 
test suite for each example automatically by applying model-based testing techniques. In particular, we 
used product line-centered test design (PLC) from our SPLTestbench as defined in ll20l . where fesfs are 
designed from fhe SPL specificafion. In confrasl fo producf-cenfered fesf approaches, where fesfs are 
designed from selecfed producf specificafions, fhe PLC approach selecfs producfs for fesfing affer fhe 
fesf design phase. This improves coverage of fhe sfafe machine, since fhe coverage criferia are applied 
onfo fhe whole SPL specificafion. 

We chose all-fransifions coverage for selecfing fhe fesfs. A fesf generafor fhen aufomafically designed 
the tests and outputs XML-documents. From the tests, SPLTestbench selected variants for testing and 
materialized them from the mutated SPL specifications into product specification mutants. 

Since our examples lack implementations, we decided to generate code from the product specification 
mutants and run the tests against them. Therefore, we developed and employed a code generator for 
transforming individual product specifications into Java. Another transformator generates executable 
JUnit code from the tests which we gained from the test generator. The mutation systems then collects 
all the code artifacts, executes the tests against the product code, and finally reports the mutation scores 
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for all tests and for every operator individually. All of the transformations above and the mutation system 
are part of our SPLTestbench. 

Generating code and tests from the same basis for testing the code is not feasible in productive 
environments, since errors propagate from the basis to code and tests. However in our case, tests are 
executed against code derived from mutated artifacts, which are different from the original. 

4.3.1 Examples 

Our examples represent three kinds of systems: an e-commerce shop (eShop), which makes heavy use 
of signals but with only few guards, a ticket machine (TicketMach) that uses less signals and in contrast 
more guards, and lastly, an alarm system (AlarmSys), which uses various signals and guards and is more 
variant-rich than the other two case studies. 

The eShop is a fictional example designed by ourselves, which is comprised of 10 features offering 
20 different variants. A customer can browse the catalog of items, or if provided, use the search function. 
Once the customer put items into the cart, he can checkout and may choose from up to three different 
payment options, depending on the eShop’s configuration. The transactions are secured by either a 
standard or high security server. A constraint ensures that credit card payment is only offered if the 
eShop also implements a high security server. 

The TicketMach example is adopted from Cichos et al. Q. The functionality is as follows: a cus¬ 
tomer may select tickets, pay for them, receive the tickets, and collect change. The feature model has a 
root feature with three sub-features attached to it; all of them are optional with no further constraints, thus 
it offers eight variants. Depending on the selected features, the machine offers reduced tickets, accepts 
not only coins but also bills, and/or will dispense change. 

From Cichos et al. Q we also adopted and extended the AlarmSys example. Currently, it consists 
of 12 features and offers 42 variants. The alarm may be set off manually or automatically by a vibration 
detector. Both features are part of an or-group and, thus, at least one of the two features must be present in 
every product. In the event of an alarm, a siren or a warning light will indicate the security breach. When 
the vibration does not stop after a predefined period of time, the system optionally escalates the alarm by 
calling police authorities and/or sending photos of evidence. Additionally to its alarming functionality, 
the AlarmSys SPL provides a feature for taking a photo of any operator that configures fhe sysfem for 
securify measures. 

4.3.2 Results 

We were able fo assess fhe fesf qualify for all fhree fesf suifes derived from fhe examples. Here, we 
presenf our resulfs. For each mufafion operafor we measured fhe amounf of defecfed mufanfs based on 
fhe SPL specification. In addifion, we assessed accumulafed mufafion scores for each example over all 
mufafion operafors and vice versa, fhe accumulafed resulfs for each mufafion operafor over all examples. 
The defailed resulfs for feafure mapping operafors can be read from Table [T] and for UML operafors from 
Table H 

Furthermore, we fracked for every example fhe number of original producfs selecfed for fesfing, 
generafed producf line mufanfs, and maferialized producf mufanfs. Tesf-wise we counfed fesfs, fesf sfeps 
by means of stimuli and expecfed reactions in all fesfs, fesfs execufed againsf all producf mufanfs, and 
fhe number of failed fesfs during fesf execution. 

For fhe eShop example, SPLTesfbench selected four producfs for fesfing. Independenf from fhis, fhe 
mufafion sysfem generafed 30 producf line mufanfs and 96 producf mufanfs for fhe mapping mufafion 
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Table 1: Mapping Operator Scores per Mu¬ 
tation Operator in % and Accumlated Scores 


(Ace) 

Op. 

eShop 

TicketMach 

AlarmSys 

Acc 

DMP 

0.00 (4) 

0.00 (5) 

0.00 (8) 

0.00 

DME 

0.00 (14) 

0.00 (8) 

0.00 (21) 

0.00 

IME 

75.00 (4) 

40.00 (5) 

50.00 (8) 

52.94 

SWP 

100.00 (4) 

60.00 (5) 

62.5 (8) 

70.59 

CFV 

100.00 (4) 

100.00 (5) 

87.50 (8) 

94.12 

Acc 

36.67 (30) 

35.71 (28) 

30.19(53) 

33.33 


Table 3: Summarized Results for Mapping Op¬ 
erators 


eShop TicketMach AlarmSys 

Products for testing 

4 

4 

6 

Product line mutants 

30 

28 

53 

Product mutants 

96 

56 

278 

Tests 

13 

9 

12 

Test steps 

103 

68 

62 

Tests executed 

302 

252 

537 

Failed Tests 

20 

30 

37 


Table 2: UML Operator Scores per Mutation 
Operator in % and Accumlated Scores (Ace) 


Op. 

eShop 

TicketMach 

AlarmSys 

Acc 

DTR 

89.29 (28) 

84.21 (19) 

63.16(19) 

80.30 

CTT 

64.29 (28) 

63.16(19) 

36.84(19) 

56.06 

DEE 

100.00(16) 

82.35 (17) 

61.54(13) 

82.61 

DTI 

82.61 (23) 

100.00(13) 

94.12(17) 

90.57 

ITG 

20.83 (24) 

27.78(18) 

16.67(18) 

21.67 

DGD 

0.00 (1) 

42.86 (14) 

50.00 (2) 

41.18 

CGD 

100.00 (2) 

68.75 (48) 

90.00(10) 

73.33 

Acc 

69.67 (122) 

66.89 (148) 

57.17(98) 

65.21 

Table 4 

: Mutation Results for State Machine 

Operators 




eShop TicketMach AlarmSys 

Products for testing 

4 

4 

6 

Product line mutants 

122 

148 

98 

Product mutants 

478 

296 

585 

Tests 


13 

9 

12 

Test steps 

103 

68 

62 

Tests executed 

1553 

1332 

1168 

Failed Tests 

283 

272 

123 


operators. For the state machine mutation operators it generated 122 product line mutants and 478 
product mutants. Every test from the 13 tests for this example were executed against every suitable 
mutant. This results in 302 test executions for the mutants created by the mapping mutation operator and 
1553 test execution for state machine mutation operators. Ultimately, 20 tests for mapping operators and 
283 tests for state machine operators failed, killing 69.67% and 36.67% of the mutants. 

Analog to the eShop, we executed less tests and generated less product mutants for the feature map¬ 
ping operators: 252 tests were executed against 56 product mutants. The tests yield an even lower 
mutation score of 35.71% than for the eShop case study. 

In case of the AlarmSys, we executed 537 tests against 278 product mutants created by the mapping 
mutation operators and 1168 tests against 585 product mutants created by the state machine mutation 
operators. Eventually, 37 and 123 tests failed, killing 30.19% and 57.17% of the mutants, respectively. 
The results are summarized in Table [3] and lU 

5 Related Work 

Mutation analysis for SPEs seems to be a rather new topic. To our knowledge, there is no publication 
dealing with mutation operators on all model artifacts of a SPE specification. Though, Henard et al. 
proposed two mutation operators for feature models based on propositional formulas in lITSl . They 
employ their mutation system for showing the effectiveness of dissimilar tests, in contrast to similar tests. 
Eor calculating dissimilarity, the authors provide a distance metric to evaluate the degree of similarity 
between two given products. 

In contrast, mutation analysis for behavioral system specifications, e.g. finite state machines, is 
established since two decades. Eabbri et al. introduced mutation operators for finite state machines 
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in 1121 . In addition to our operators, they also consider adding states and the exchange of elements 
(event, guard, effect) by another. Belli and Hollmann provide mutation operators for multiple formalism: 
directed graphs, event sequence graphs lO, finite state machines Il25l . and basic state charts |3|. They 
conclude, that there are two basic operations from which most operations can be derived: omission and 
insertion. Also for timed automata, mutation operators can be found in |[T1. 

In |[32l Stephenson et al. propose the use of mutation testing for prioritizing test cases from a test 
suite in a SPL environment. Unfortunately, the authors provide no evaluation of their approach. 


6 Conclusions 

In this contribution, we lifted mutation analysis to the product line level. We defined and invesfigafed 
mufafion operafors for feafure models, mapping models, and UML models. As opposed fo producf-based 
mufafion analysis, our mufafion operafors are based on fhe SPL specification. This allows us fo mimic 
realisfic errors made by humans during modeling a SPL. To our knowledge, fhis is fhe firsf sfep fowards 
a qualifafive evaluafion of SPL fesfs, which is based on fhe SPL’s specificafion. 

Our resulfs for fhe fhree examples are as expecfed for mosf of fhe mufafion operafors. As predicfed, 
mufafion operafors confribufing superfluous behavior are hard fo defecf for conformance fesfs. Such 
mufafions are DMP (0%) and DME (0%) on feafure mappings and ITG (21.67%) on domain models. 
For mosf of fhe ofher operafors we gain scores above 70%, which is in fhe expecfed range for all- 
fransifions coverage ll^ . For DGD and CTT mufafions fhe fesfs score surprisingly low resulfs. Here, 
furfher invesfigafions seem necessary. 

In conclusion, we idenfified a lack of error defection capabilify in sfandard fesf procedures for SPFs. 
Even simple errors are nol defecfable, neifher by all-fransifions, MC/DC as for safefy-crilical system, 
nor any ofher conformance fesf procedure. As indicated, fhe resulfs are applicable fo af leasf fhe here 
surveyed SPF engineering paradigms negafive/posifve variabilify and delfa modeling. We assume, ofher 
paradigms suffer from fhis lack as well. Unforfunafely, currenf procedures for negative fesfing, which 
could pofenfially defecf such errors are still nof enabled for SPFs. Thus, fufure work will proceed fo 
enable negative fesfing procedures for SPFs. 

In |[20l . we described producf-cenfered and producf line-centered fesf design processes for SPFs. We 
plan fo employ fhis mufafion sysfem for assessing fhe qualify of fhe fesf suifes generafed by fhe differenf 
fesf design mefhods. From fhe resulfs we hope fo gain general directions towards favorable fesf design 
mefhods and processes by means of error defection capabilify, fesf efforf, and efficiency. 

Furfhermore, we wanf fo investigate higher-order mufafion operafors, fhaf combine more fhan one 
change af a fime fo a producf. For fhis purpose we need co-adapfafions, so fhaf fhe parts fhaf consfifufe fhe 
SPF, here feafure model, mapping/della model, and domain model, are adapted fo preserve consistency 
when one of fhe parfs changes. For example, such an adapfafion is necessary affer fhe delefion of a feafure 
fo ensure fhaf fhere is no broken feafure reference in related mappings. In |[^ . we presenfed a profofype 
for co-adapfafions in anofher model-based scenario, namely domain-specific language developmenf. 


Acknowledgments 

This work is supporfed by granfs from Deufsche Forschungsgemeinschaff, Graduierfenkolleg METRIK 
(GRK 1324). 


70 


Potential Errors and Test Assessment in Software Product Line Engineering 


References 

[1] Bernhard K. Aichernig, Florian Lorber & Dejan Nickovic (2013): Time for Mutants — Model-Based Mu¬ 
tation Testing with Timed Automata. In David Hutchison et ah, editor: Tests and Proofs, Lecture Notes in 
Computer Science 7942, Springer Berlin Heidelberg, pp. 20-38, doi: 10.1007/978-3-642-38916-0J2 

[2] Fevzi Belli, Christof J. Budnik & Lee White (2006): Event-based modelling, analysis and testing of user 
interactions: approach and case study. Software Testing, Verification and Reliability 16(1), pp. 3-32, 
doi:10.1002/stvr.335 

[3] Fevzi Belli & Axel Hollmann (2008): Test generation and minimization with basic statecharts. In Edward J. 
Delp & Ping Wah Wong, editors: the 2008 ACM symposium, vol. 5681, SPIE and IS&T, Bellingham and 
Wash and Springfield and Va, p. 718, doi: 10.1145/1363686.1363856 

[4] Eevzi Belli, Axel Hollmann & Sascha Padberg (2011): Model-Based Integration Testing with Communication 
Sequence Graphs. In Justyna Zander et al., editor: Model-based testing for embedded systems. Computa¬ 
tional analysis, synthesis, and design of dynamic systems, CRC Press, Boca Raton, doi: 10.1201/bl 1321-10 

[5] Paul E. Black, Vadim Okun & Y. Yesha (2000): Mutation operators for specifications. In: ASE 2000 15th 
IEEE International Automated Software Engineering Conference, pp. 81-88. doi: 10.1109/ASE.2000.873653 

[6] Harald Cichos & Thomas S. Heinze (2011): Efficient Reduction of Model-Based Generated Test Suites 
Through Test Case Pair Prioritization. In: Proceedings of the 7th International Workshop on Model- 
Driven Engineering, Verification and Validation, IEEE Computer Society Press, Los Alamitos, pp. 37^2. 
doi:10.1007/978-3-642-21210-9_24 

[7] Harald Cichos, Malte Lochau, Sebastian Oster & Andy Schiirr (2012): Reduktion von Testsuiten fur 
Software-Produktlinien. In Stefan Jahnichen et al., editor: Software Engineering 2012: Eachtagung des 
GI-Eachbereichs Softwaretechnik, 27. Eebruar - 2. Marz2012 in Berlin, ENI 198, GI, pp. 143-154. 

[8] Paul Clements & Linda Northrop (2009): Software product lines: Practices and patterns, 7. print edition. 
The SEI series in software engineering, Addison-Wesley, Boston Mass. u.a. 

[9] Krzysztof Czarnecki & Michal Antkiewicz (2005): Mapping Features to Models: A Template Ap¬ 
proach Based on Superimposed Variants. In Robert Gliick, editor: Generative programming and com¬ 
ponent engineering, Eecture Notes in Computer Science 3676, Springer, Berlin [u.a.], pp. 422^37, 
doi: 10.1007/11561347_28, 

[10] Richard A. DeMillo (1980): Mutation Analysis as a Tool for Software Quality Assurance. In: COMPSAC’80. 

[11] Emelie Engstrom & Per Runeson (2011): Software product line testing -A systematic mapping study. Infor¬ 
mation and Software Technology 53(1), pp. 2-13, doi: 10.1016/j.infsof.2010.05.011 

[12] Sandra C. P. E Eabbri, M. E. Delamaro, J. C. Maldonado & P. C. Masiero (1994): Mutation analysis testing 
for finite state machines. In: 1994 IEEE International Symposium on Software Reliability Engineering, pp. 
220-229. doi: 10.1109/ISSRE.1994.341378 

[13] Iris Groher & Markus Voelter (2007): Expressing Feature-Based Variability in Structural Models. In: Work¬ 
shop on Managing Variability for Software Product Lines. Available at http: //www. voelter. de/data/ 
workshops/MVSPL_GroherVoelter.pdf 

[14] Hans Gronniger, Holger Krahn, Claas Pinkernell & Bernhard Rumpe (2008): Modeling Variants of Automo¬ 
tive Systems using Views. In Thomas Kiihne, Wolfgang Reisig & Eriedrich Steimann, editors: Tagungsband 
zur Modellierung 2008 (Berlin-Adlershof, Deutschland, 12-14. Marz 2008), LNI, Gesellschaft fiir Infor- 
matik, Bonn. Available at http: //arxiv. org/abs/1409.6629 

[15] Christopher Henard, Mike Papadakis, Gilles Perrouin, Jacques Klein & Yves Le Traon (2013): Assessing 
Software Product Line Testing Via Model-Based Mutation: An Application to Similarity Testing. In: ICSTW 
’13: IEEE 6th International Conference On Software Testing, Verification and Validation Workshops 2013, 
pp. 188-197. doi: 10.1109/ICSTW.2013.30 

[16] Yue Jia & Mark Harman (2009): Higher Order Mutation Testing. Inf. Softw. Technol. 51(10), pp. 1379-1393, 
doi:10.1016/j.infsof.2009.04.016 



Hartmut Lackner & Martin Schmidt 


71 


[17] Yue Jia & Mark Harman (2011): An Analysis and Survey of the Development of Mutation Testing. IEEE 
Transactions on Software Engineering 37(5), pp. 649-678, doi: 10.1109/TSE.2010.62, 

[18] K. C. Kang, S. G. Cohen, J. A. Hess, W. E. Novak & A. S. Peterson (1990): Feature-Oriented Domain 
Analysis (FODA) Feasibility Study. Available at http: //www. sei . emu. edu/reports/90tr021 .pdf 

[19] S. Kim, John A. Clark & J. A. Mcdermid (2000): Class Mutation: Mutation Testing for Object-Oriented 
Programs. In: EMES. 

[20] Hartmut Lackner, Martin Thomas, Elorian Wartenberg & Stephan WeiBleder (2014): Model-Based Test De¬ 
sign of Product Lines: Raising Test Design to the Product Line Level. In: ICST’ 14: International Conference 
on Software Testing, Verification, and Validation, pp. 51-60. doi: 10.1109/1CST.2014.16 

[21] Make Lochau, Ina Schaefer, Jochen Kamischke & Sascha Lity (2012): Incremental Model-Based Testing of 
Delta-Oriented Software Product Lines. In David Hutchison et al., editor: Tests and Proofs, Lecture Notes 
in Computer Science 7305, Springer Berlin Heidelberg, Berlin and Heidelberg, pp. 67-82, doi: 10.1007/978- 
3-642-30473-6_7 

[22] Object Management Group (OMG) (2011): UML 2.4.1 Superstructure Specification. 

[23] Jeff Offutt (1992): Investigations of the Software Testing Coupling Effect. ACM Trans. Softw. Eng. Methodol. 
1(1), pp. 5-20, doi: 10.1145/125489.125473 

[24] Jeff Offutt, Shaoying Liu, Aynur Abdurazik & Paul Ammann (2003): Generating test data from state- 
based specifications. The Journal of Software Testing, Verification and Reliability 13, pp. 25-53. 
doi:10.1002/stvr.264 

[25] Jeff Offutt, Shaoying Liu, Aynur Abdurazik & Paul Ammann (2003): Generating test data from state- 
based specifications. The Journal of Software Testing, Verification and Reliability 13, pp. 25-53. 
doi:10.1002/stvr.264 

[26] Jeff Offutt & Roland H. Untch (2001): Mutation 2000: Uniting the Orthogonal. In W.Eric Wong, editor: Mu¬ 
tation Testing for the New Century, The Springer International Series on Advances in Database Systems 24, 
Springer US, pp. 34-44, doi: 10.1007/978-l-4757-5939-6_7 

[27] Erika Mir Olimpiew & Hassan Gomaa (2005): Model-Based Testing for Applications Derived from Software 
Product Lines. ACM SIGSOET Software Engineering Notes 30(4), pp. 1-7, doi: 10.1145/1082983.1083279 

[28] Sebastian Oster, Ivan Zorcic, Elorian Markert & Make Lochau (2011): MoSo-PoLiTe: tool sup¬ 
port for pairwise and model-based software product line testing. In: VaMoS ’ll, pp. 79-82. 
doi: 10.1145/1944892.1944901 

[29] Klaus Pohl, Gunter Bockle & Linden, Erank J. van der (2005): Software Product Line Engineering: Founda¬ 
tions, Principles and Techniques. Springer-Verlag New York, Inc, Secaucus and NJ and USA. doi: 10.1007/3- 
540-28901-1 

[30] Ina Schaefer (2010): Variability Modelling for Model-Driven Development of Software Product Lines Sys¬ 
tems, Linz, Austria, January 27-29, 2010. Proceedings. In D. Benavides, D. Batory & P. Griinbacher, editors: 
Fourth International Workshop on Variability Modelling of Software-Intensive Systems, Linz, Austria, Jan¬ 
uary 27-29, 2010. Proceedings, ICB-Research Report 37, Universitat Duisburg-Essen, pp. 85-92. 

[31] Martin Schmidt, Arif Wider, Markus Scheidgen, Joachim Eischer & Sebastian von Klinski (2013): Refactor¬ 
ings in Language Development with Asymmetric Bidirectional Model Transformations. In Eerhat Khendek, 
Maria Toeroe, Abdelouahed Gherbi & Rick Reed, editors: SDL 2013: Model-Driven Dependability Engi¬ 
neering - 16th International SDL Eorum, Montreal, Canada, June 26-28, 2013. Proceedings, Lecture Notes 
in Computer Science 7916, Springer, pp. 222-238. doi: 10.1007/978-3-642-38911-5_13 

[32] Zoe Stephenson, Yuan Zhan, John Clark & John McDermid (2004): Test Data Generation for Product Lines 
- A Mutation Testing Approach. In Birgit Geppert et al., editor: SPLiT ’04: Proceedings of the International 
Workshop on Software Product Line Testing, Boston and MA, pp. 13-18. 

[33] Stephan WeiBleder (2009): Influencing Factors in Model-Based Testing with UML State Machines: Report 
on an Industrial Cooperation. In David Hutchison et al., editor: Model Driven Engineering Languages and 




72 


Potential Errors and Test Assessment in Software Product Line Engineering 


Systems, Lecture Notes in Computer Science 5795, Springer Berlin Heidelberg, Berlin and Heidelberg, pp. 
211-225, doi: 10.1007/978-3-642-04425-0_16 

[34] M. R. Woodward (1993): Errors in algebraic specifications and an experimental mutation testing tool. Soft¬ 
ware Engineering Journal 8(4), p. 211, doi: 10.1049/sej. 1993.0027 


